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Abstract: This paper deals with issues related to drawing up plans and putting into practice e-learning initiatives 
in complex and demanding businness environments such as organizations that require recurring training, to 
optimize their learning processes in terms of cost reduction and efficiency gains. We view the issue of initiating 
and executing such an e-learning initiative both by the “process” and “product” perspective. 

Towards this end, our approach consists of conceptual tools and proposes a way of working supporting: 

■ Depicting and tracking personal competencies facilitating capturing of individuals’ informal 
knowledge and implicit and explicit evaluation 

■ Providing dynamically optimal recommendations to learners for individual learning paths, 
learning content and learning programmes 

We propose that in order to achieve competency based delivery of learning, solutions should imply a learner 
centric apprach taking under consideration the formal and informal knowledge that learmers posses. Going a step 
further, we present a first attempt of modeling the whole evaluation cycle from the initial set up of required 
proffesional profiles and how this is tied to a given work position, to the final optimal recommendations that a 
solution should provide/deliver based on personalised evaluation of the competencies that learner has achieved 
or he/she wants to obtain. 
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1. Introduction 

This paper aims at providing an approach for 
facilitating the design process of mainly large- 
scale e-learning acts with the purpose of 
serving the requirements and needs of 
organizations with geographically distributed 
employees and short learning curves, for 
enhanced learning delivery based on the 
competencies that their employees posses as 
well as on the anticipated training level they 
need to obtain. Customized and track learning 
on individual participant basis, and flexibility 
greater than the one offered in traditional 
Learning is one of the major goals of our 
research. 

We present a framework for supporting large 
scale e-learning initiatives in big distributed 
and highly demanding organisations by 
viewing the issue from two different 
perspectives: 

■ the “process” perspective: the initiative 
set-up and execution i.e. the way of 
working towards an e-learning initiative 
lifecycle (goal setting, requirements 
elicitation, analysis and design, 
implementation, feedback and so on) 

■ and the “product”: the learning delivery 
to the final-users i.e. the final outcome of 
the process and its quality (personalized 
learning, quality of learning etc.) 

The proposed framework will entail from the 
process viewpoint 


■ a set of well documented conceptual 
tools for the analysis of the professional 
knowledge and skills and of the 
production of professional profiles in the 
working environment for identified 
specialties. 

■ A consistent methodology for the 

evaluation and the professional 
improvement through educational 
processes, on the basis of the 
knowledge and skills of each employee, 
as well as 

■ A way of working for applying the 

proposed conceptual tools in real life 
projects, 

while form the product viewpoint it tackles 
issues of personalised learning improved in 
quality. 

Personalised learning will be achieved by 
enabling efficient evaluation of the employees 
using “traditional” evaluation techniques (on¬ 
line evaluation) as well as knowledge 

management techniques (tracing of learning 
behaviour patterns and informal knowledge) 
[Leo 19971 , [Jimes 20031 This evaluation leads 
to the definition of workers’ basic deficiencies, 
as regards skills and knowledge (gap analysis) 
in relation to a required professional profile for 
a given job-position. The coverage of the 
detected deviations will be supported by 
dynamic recommendations for the subsequent 
learning process progress [Petropoulos, Xini 
20031 . Domain specific and language 
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independent taxonomies are defined and 
employed in order to map the training subject 
to the required job profiles that need to be 
obtained through the training sessions as well 
as for the appropriate recommendations, which 
will be dynamically directed to the learner. 

The rest of the paper is organised as follows: 
Section 3 presents the background of this 
research and the motivation we had for 
tackling competency based e-learning delivery. 

In Section 4 we describe the learner centric 
approach for competency based learning 
delivery by defining the coresponding 
processes 

Section 5 deals with the appropriate evaluation 
methodological framework for the evaluation of 
the knowledge and skills of the employees, 
adapted to the frame of individual companies’ 
requirements. 

Then section 6 goes in detail describing and 
sketching a conceptual architecture for the 
“product” perspective 

Finally section 7 presents the conclusions 
drawn from the work presented and proposes 
topics and steps for remaining and future work. 

2. Background 

The motivation for the work described in this 
paper and the methodology that was 
developed came as a straightforward 
requirement from research and development 
projects in different market sectors that our 
company is involved in. The approach followed 
by many of our customers, medium or larger 
organizations, with respect to e-learning 
services targeted to their personnel is rather 
technology oriented, focused primarily on 
providing the necessary technology platform 
and ensuring participation of their employees - 
trainees. They rely heavily on the functionality 
and evaluation characteristics that proprietary 
solutions offer. This in turn yields a 
misconception of the derived training results as 
well as a lack of methodological tools in order 
to measure efficiency gains, cost savings, 
individual professional improvement, groups’ 
performance enhancement, and organization 
advancement [Davenport, Prusak 19981 . 

Retroactive analysis of the different cases led 
us to establish a set of criteria for measuring 
the level of achievement of the objectives of a 
total competency based learning delivery 
solution from the perspective of personalized 
delivery of knowledge, human resources 
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continuous evaluation, evolution tracking and 
improvement. This analysis was mainly based 
on the study of the deficiencies as well as the 
strong points of a variety of project cases. 

The criteria set were: 

■ Personalisation'. If the systems offer 
services of personalized learning, 
access to knowledge and the degree 
that the system offers these services. 
We also examine whether the system is 
in position to construct, maintain and 
update a personal learning profile in a 
formal way, which is going to be 
consulted every time a knowledge 
intensive task is performed. 

■ Evaluation : Whether the system offers 
explicit or implicit evaluation 
mechanisms (on line tests, quizzes 
versus tracking of behavior, intelligent 
mechanisms). 

■ Evolution tracking : With this criterion we 
examine whether the solution under 
consideration captures and measures 
the progress of the individual learner 
within the learning environment using a 
formal way. 

■ Human Resources (HR) support : This 
criterion is highly interconnected with the 
previous one, and it is defined for 
examining whether the e-learning 
solution is in position to provide valuable 
and exploitable information concerning 
the trainees (e.g. evaluation) as well as 
useful directions for HR development 
(identified gaps of the staff, suggestions 
for knowledge provisions-training, 
knowledge assets etc.) 

3. Competency based learning 
delivery 

We propose a learner centric environment as 
schematically represented in Figure 1 below.: 

In order to achieve dynamic evaluation of the 
employ’s learning paths according to the skills 
he/she possesses and the required ones that 
are the actual target of the training, a learner 
centric approach is proposed comprising as 
building blocks four main elements: 

1. The existing profile of the learner: It 
describes the employees’ skills and 
current employment tasks 

2. The Required Profile: It describes 
formally the skills that should be 
developed, enhanced or updated 
according to the training plan 

3. Explicit Evaluation Results: These are 
the actual results that the employee 
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achieves through normal evaluation of 
tests and learning exercises 

4. Learning Behaviour Patterns: These are 
the patterns that the employee 
generates in the learning process 
involved. 


These four elements represent the 
corresponding conceptual entities that are 
captured and then processed in the proposed 
environment. 


Existing Profile 


Explicit 

Evaluation 



Evaluation 

Engine 


Figure 1: Learner centric approach 


3.1 Process definition 

Initially, a sampling research is suggested with 
structured questionnaires and processing of 
the filled questionnaires for example through 
the use of multi-criteria analysis, in order to 
define the professional profiles [Siskos et.al. 
20001 . 

The sampling research (apart from the general 
nature data) shall focus into two basic sectors: 

■ To the definition of general professional 
profiles, and 

■ To the evaluation and modeling of the 
human force on the basis of objective 
criteria 

Collected data constitute the initial input and 
thet are then processed using the 
methodological approach of multi-criteria 
analysis for the assesment of criteria and 
indicators [Mendoza et.al. 19991 . This analysis 
supports the definition of the satisfaction 
indicators of the employees regarding their 


professional role, as well as their objective 
evaluation on the part of their companies. 

With the aid of these indicators, the degree 
(grade) of overall satisfaction, and the 
satisfaction indicator of the individual factors 
are evaluated. 

Given that the most important criteria for the 
placement and choice of personnel for the 
employers today are the “competence of 
knowledge”, and the “professional experience”, 
these elements should be verified, along with 
the degree of response to the market needs. 

The collected and analysed data should be 
modeled to form the conceptual basis for 
susequent use. Several approaches may be 
employeed for the modelling task such as 
Enterprise Knowledge Development tools 
[Loucopoulos et.al. 19971 . 

Among the general professional profiles that 
are generated, some will be selected and 
adapted to the needs of the company in order 
to define the degree of compatibility with the 
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requirements of the market. These profiles are 
categorised based on the specialties that the 
learning initiative aims to train. In this way two 
different models for the selected categories of 
profiles are generated, consisting of 
characteristics, knowledge and skills for each 
selected professional specialty. These, we 
conventionally call «ideal professional profile» 
and «required professional profile» in contrast 
to the real professional profile of each 
employee and are both compliant with IMS 
Reusable Definition of Competency or 
Educational Objective RDCEO information 
model riMS-RCDEO(a)20Q21 

The evaluation cycle starts on the basis of the 
detection of knowledge and skills of each 
employee (definition of the real professional 
profile) in relation to the required profile of its 
category. This evaluation constitutes the basic 
pre-evaluation for the classification of the 
employee in some professional profile. The 
procedure of the evaluation is implemented 
and ends up whether in the detection and 
certification of a gap between the required and 
real professional profile, or in the interruption 
of the procedure. The interuption occurs either 
in the case that the employee is considered 
insufficient in relation to the characteristics of 
his role and must be placed in another 
position, or in case the evaluation results show 
that the employee is under-employed in 
relation to his qualifications and possibly he 
must be granted a higher level of work. 

The definition and analysis of the gap should 
be implemented continuously through 
electronic applications for evaluation, 
specialized data (positions requirements - 
position owners), processing of the data, 
informal knowledge tracing mechanisms 
(learners behaviour patterns, taxonomies, 
reasoning), classification of the data, grouping 
of the personnel according to the differences, 
representation of the results of processing and 
other similar techniques so as to produce 
utilizable and valid results. 

The management of the differences that may 
occur, being positive or negative, will be 
conducted on the basis of the required 
professional role or the future professional 
improvement of the employee through the 
provision of training programmes. This training 
will aim to provide to the learner 
supplementary knowledge and skills in order to 
approach the required professional profile 
tailored for the respective job position. 


http://www.eiel.org 


The whole procedure should be manageable 
by a sufficient toolset, that does not only 
evaluate the present knowledge level of the 
employees, but also use knowledge 
management techniques in order to chose the 
most appropriate among the available training 
sources for the implementation of the required 
professional profile. 

In this way a dynamic planning of an ad-hoc 
«personalized» program for the development 
of the human resources is generated. In turn, 
this leads to overall management of knowledge 
and skills, at the organisation level as well as 
at the level of the knowledge background of 
the employees, resulting to the faster and 
more effective coverage of the existing working 
positions, the rapid management of human 
resources and the more effective management 
of the professional evolution of the employees. 

The overall procedure «closes» with the final 
evaluation, that will lead whether to the 
planning of future required professional 
characteristics of the employee, in case of 
success, or to the re-introduction of the 
employee in the process in case that the 
deviation between required and real 
professional profile persists. 

In the frame of the previous analysis of the 
evaluation system, the following are involved 
accumulatively: 

■ Management of the data related to 
working position (role, tasks, actions, 
communications, demands, etc.). 

■ Management of the CV data and 
profiling data of the personnel 
(education, training, experience, skills) 
[EURES 20031 . 

■ Definition of the gap between the 
present situation and the required one 
(as regards knowledge, experience and 
skills of human resources). 

■ Monitoring of the evolution / 
development of the human resources. 

At the same time, the evaluation focuses also 
to the dynamic and effective management of 
parameters like: 

■ Classification of the training into 
categories and levels 

■ Training resources availability. 

■ Creation of training improvement 
programs by combining existing training 
units. 

■ Indicators of personal and professional 
satisfaction of employees and 
companies. 
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Having in mind the abovementioned data and 
in order to be able to implement the IT tools 
that can support the overall approach, the 
proposed actions are the following: 

■ Analysis of knowledge and skills 
according to the professional profiles 
selected 

■ Classification of knowledge in distinct 
parts of the knowledge subject 

■ Connection of skills with particular 
measurable results that the employee 
must achieve whether during the training 
or during work 

■ Creation of data collections with 
categories of questions, exercises, case 
studies and papers that will be 
presented or assigned to the employee 
according to the evolution of the training, 
during training 

■ Evaluation of the answers to the 
questions- exercises or the results 
achieved in case studies and work 
assignments 

■ Focusing on several questions 
categories, exercises categories, case 
studies and work assignments. 

■ Tracing and analysis of informal 
knowledge, i.e. knowledge not declared 
in the employees’ formal CV. 

The deducing mechanism of the proposed 
evaluation information tools will have input 
data derived from the training process of the 
employee, as well as data derived from his 
professional evolution and working and 
learning behaviour. Thus with the proper 
combinations it is feasible to define the level of 
knowledge and skills of the employee as 
compared to the required professional profile. 

4. Evaluation methodology 

4.1 Personnel’ skills 

The proposed approach uses a consistent 
methodological framework for the evaluation of 
the knowledge and skills of the employees- 
learners, adapted to the frame of individual 
companies’ requirements. The framework 
provides the mechanisms enabling evaluation 
of the employees knowledge, and individalised 
paths for upgrading the knowledge and skills of 
the employees, according to the defined 
requirements of their professional role. 

4.2 Gap analysis 

An integral part of the methodology is based 
on the evaluation of the gap between the 
requirements of the role of the working 


positions and the competencies (knowledge, 
training, experience, skills) in workers 
occupying these working positions. This Gap 
Analysis leads to the following: 

■ Estimation of the degree of knowledge, 
experience and skills (competency 
analysis) of the workers in relation to the 
requirements of their role - job 
description and their objectives for 
personal evolution (within the working 
settings) 

■ Definition of the required education / 
training 

■ Monitoring of the learning process and 
the “behaviour” of the learner 

4.3 Evaluation cycle 

This information provides the specified 
approach of a dynamic adaptation of the 
employees to the requirements of their 
professional role and consequently of the more 
general management of learners’ knowledge. 

The described procedure is recursive, aiming, 
on the one hand, to the better utilization of the 
human resources on the part of the company 
and, on the other, to the global satisfaction of 
the employees purposes regarding their 
continuous professional improvement. 

Finally, the total evaluation is conducted on the 
basis of three distinct but fully supplementary 
levels, i.e.: the employability of the employee, 
the satisfaction of the employee’s goals in 
connection with his professional evolution 
derived from the attendance of the training 
program, as well as from the total evaluation of 
the employee and his professional evolution by 
the employer company. 

Evaluation is divided in two categories: 

Explicit Evaluation: These are the actual 
results that the employee achieves through 
normal evaluation of tests and learning 
exercises (LMS Evaluation tests, exercises, 
formal trainees’ evaluation results etc.) 

Implicit Evaluation based on the informal 
knowledge traced from his learning behavior 
and on-the job processes that he/she is 
carrying out, strictly related to the training 
subject (Learning Behaviour Patterns). 

Diagrammatically, the frame of Estimation and 
Evaluation of the Employees Knowledge and 
Skills is presented in Figure 2. 


http://www.eiel.org 


©Academic Conferences Limited 



Electronic Journal on e-Learning Volume 2 Issue 1 (February 2004) 227-236 


232 



Figure 2: Employees Knowledge and Skills Evaluation Cycle 


5. Conceptual architecture 

The product perspective is assessed with a 
proposed conceptual architecture depicted in 
Figure 3. The conceptual architecture is an 
attempt to provide high-level view of an 


integrated solution that performs the required 
processes of the evaluation framework. It 
presents the main functional components that 
can achieve the competency based learning 
delivery to the end-users, the learners. 



RDF Domain 
I Specific Taxonomies 


Tools 


Communication 

Tools 


Figure 3: Conceptual architecture 


Our approach considers a main precondition 
■ There are Proprietary KM 
Environments (PKME) in the target 
organisation, which usually play 


specific role in a training process. 
These can consist of Learning 
Management Systems, KM Supporting 
tools, Intranets, Organisations’ file 
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servers, Communication and 
Collaboration tools, a combination of 
them and several other applications 
that the organisation may have in its 
possession. The data generated by 
these PKMEs may play significant role 
in the implicit evaluation of the 
learners’ training process and usually 
are strictly connected with their tacit 
knowledge. 

5.1.1 Sources adaptors 

In order to trace the learners’ behaviour during 
their training process, our approach considers 
all potential internal and external sources that 
the learner uses as the depicted Proprietary 
KM Environments (PKME). 

The PKMEs of an organisation used during a 
training process are generating data according 
to the learners’ usage behaviour and their 
training patterns. Those data may well 
describe the tacit knowledge of learners 
generated through the training process and are 
considered as preliminary knowledge entities 
that need further processing in order to add 
value in the evaluation methodology. 

The role of sources adaptors is to provide the 
appropriate interfaces for Knowledge Feeder in 
the process of collecting specific data related 
to learners training process strictly connected 
with their informal knowledge. 

We differentiate between two types of sources 
with respect to the adaptors: 

■ proprietary applications/tools or 
content sources that comply with 
standards as for example a Learning 
Management System compliant with 
LIP standard that the organisation may 
possess riMS-LIP (2001)1 . 

■ Applications or tools that may not be 
standards compliant, usually 
applications and tools developed in 
house or that serve individual and/or 
specific organisation’s needs. 

In the first case adaptors are set in order to 
collect data elements from PKMEs in a certain 
structure while in the second, adaptors have to 
be built in order to provide the relevant data in 
a specified standard compliant format. 

5.1.2 Knowledge bases 

These form an integral part of the proposed 
environment and are consisting of LMS 
repositories, Human Resources repositories, 
learning content repositories, 
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competency/skills and other relevant RDCEO 
[IMS-RDCEO(a)1 compliant repositories. 

Moreover they are including registries that are 
used by discrete components in the form of 
native XML databases such as a registry for 
the publishing and description of Knowledge 
Feeder Web Services and the RDF registry for 
the Domain Specific Taxonomies. 

5.1.3 Knowledge Feeder 

The role of Knowledge Feeder is triple. It 
collects existing evaluation data, it collects 
learners' data from the knowledge bases and 
the most crucial it collects usage data from the 
PKMEs through the specific adaptors. 
Knowledge feeder may consist of a set of Web 
Services for these three different categories 
and implements their choreography in order to 
provide the appropriate output to the 
Evaluation Engine. 

It is the main component that interacts with the 
interface and interacts also bi-directionally with 
the Evaluation Engine and Recommender 
components. 

5.1.4 Evaluation Engine 

The Evaluation Engine forms the core engine 
of the environment. Its role is to provide implicit 
and explicit evaluation results according to the 
proposed evaluation methodology in a way 
that learning gaps and deviations from learning 
paths are identified in an adaptive and 
progressive manner. The evaluation engine 
implements the core algorithms for the 
evaluation process: 

(i) It receives the test results, exercises and 
training performance evidence from the 
Knowledge Feeder and performs the 
explicit evaluation 

(ii) It receives from the Knowledge Feeder 
data representing learner’s behaviour 
and performs light reasoning with 
respect to the relevance/irrelevance of 
learner's behaviour patterns to the 
training subject. This is achieved by 
identification of Subject categories in 
these patterns and their mapping to 
predefined domain specific taxonomies. 
It proceeds with distance calculations 
and uses the outcomes for performing 
the learner’s implicit evaluation. 

(iii) It combines Implicit and Explicit 
Evaluation results with retrieved content 
from ad-hoc or existing 
competency/skills repositories and 
correlates with the required profile 
and/or the training subject goals. 
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(iv) It implements and performs the required 
Gap Analysis 

The Evaluation Engine interfaces with 
Knowledge Feeder in order to get the 
appropriate input and also for calling specific 
Web Services needed during the Evaluation 
procedure 

It provides as output to the Recommender 
component the Evaluation and Gap Analysis 
results. 

5.1.5 Recommender 

The Recommender component has the role to 
identify potential corrective activities that the 
learner should take in order to fulfil the training 
process goals. 

It receives input from the Evaluation Engine 
and interfaces with the Knowledge Feeder for 
retrieving learning objects [ARIADNE 20031 
and/or Internal or External Sources that have 
to be provided to the learner as a 
consequence of the Evaluation and Gap 
Analysis results. The identified internal and 
external sources are passed to the interface 
forming a close feedback loop for adaptive and 
progressive learning process assessment. 

5.1.6 System Interface 

The interface implements the learners’ 
interface to the whole environment as well as it 
implements the service request client for the 
Knowledge Feeder Web Services 

6. Conclusions and future work 

In this paper we attempted to present a 
framework for supporting e-learning initiatives 
by viewing the issue from two different 
perspectives i.e. the “process”: the project set 
up and execution and the “product”: the 
learning delivery to the final-users. From the 
process viewpoint we suggested a way of 
working towards setting up an e-learning 
solution together with a set of conceptual tools 
in the direction of enhancement of learners 
profiling and feedback mechanisms in a 
recursive process. From the “product” 
viewpoint we proposed a conceptual 
architecture that takes advantage of informal 
knowledge tracing mechanisms, achieves 
enhanced evaluation and provides personal 
learning paths. While we attempt to provide a 
complete and integrated approach during our 
research several research issues have been 
raised which constitute the basis of further 
study, analysis and research. Such issues 
which are still open for research are: the 


approach for modeling professional profiles i.e. 
What makes a knowledge modeling approach 
appropriate for modeling such initiatives?; the 
mechanisms of tracing informal knowledge i.e. 
Is capturing of tacit knowledge feasible? How 
can we approach such a necessity? 

We intend to apply the proposed approach in a 
series of ongoing related projects. These case 
studies will provide the evidence needed to 
evaluate the methodology per se and identify 
the potential for further improvement and 
advancements. Comparative results from 
different application domains may also provide 
insights for peculiarities and characteristics 
that exist in the individual learner's level and 
are totally hidden or difficult to extract when 
assessing a solution for a specific industry 
sector. 
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